<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Cycloctopus: A Screen Scraping Console</title>
<link rel="stylesheet" type="text/css" href="handout.css" />
<link rel="stylesheet" type="text/css" href="custom.css" />
<body>
<div id="info">
    <div id="title">Cycloctopus: A Screen Scraping Console</div>
    <div id="author">Feihong Hsu</div>
    <div id="venue">Chicago Python Users Group</div>
    <div id="location">The Fixx (Chicago, IL)</div>
    <div id="date">May 8, 2007</div>
</div>
<div class="section">
<h1>What Is Cycloctopus?</h1>
<p>Cycloctopus helps you visually explore the DOM of a page. It is mainly inspired by Kodos and Aardvark.</p>
<p>The name of the project comes from the titular character of the short story "Aw $#!*, It's Cycloctopus!" by Jeffrey Brown.</p>
<div class="notes">
<p><a class="reference" href="http://kodos.sourceforge.net/">Kodos</a> is a regular expression debugger. <a class="reference" href="http://karmatics.com/aardvark/">Aardvark</a> is a Firefox extension that helps you inspect and manipulate web pages.</p>
<p><a class="reference" href="http://www.amazon.com/Project-Superior-Various/dp/0972179488">Project: Superior</a> is the comics anthology that contains the Cycloctopus story. Many of the other stories in this collection are also quite refreshing.</p>
</div>
</div>
<div class="section">
<h1>A Demo of Cycloctopus</h1>
<p>Imagine a brilliant demo:</p>
<ul class="simple">
<li>Given by a foxy lady</li>
<li>Wearing fishnet stockings</li>
<li>Underneath her sweatpants</li>
</ul>
<div class="notes">
<p>Unfortunately, the actual demo was given by:</p>
<ul class="simple">
<li>A bewildered Asian man</li>
<li>Wearing a green fleece,</li>
<li>Obsessing about <a class="reference" href="http://www.youtube.com/watch?v=FGzf--OLoTI">light-saber-dueling gynmasts</a></li>
</ul>
</div>
</div>
<div class="section">
<h1>Features</h1>
<ul class="simple">
<li>Select DOM elements and view their tag info</li>
<li>Enter queries to select multiple elements</li>
<li>Display text node queries</li>
</ul>
<div class="notes">
<p>Two types of queries are supported: CSS and XPath.</p>
<p>When you hover over a DOM element, you can see:</p>
<ul class="simple">
<li>Tag name</li>
<li>ID</li>
<li>Classes</li>
<li>Depth</li>
<li>Number of children</li>
<li>Path (relative to the body tag)</li>
</ul>
<p>Text nodes cannot be selected (not visually, anyway). So Cycloctopus displays text nodes by opening a separate dialog that lists the contents of all the text nodes that were returned.</p>
</div>
</div>
<div class="section">
<h1>Key Controls</h1>
<table border="1" class="key-table docutils">
<colgroup>
<col width="29%"/>
<col width="71%"/>
</colgroup>
<tbody valign="top">
<tr><td><tt class="docutils literal"><span class="pre">w</span></tt></td>
<td>Widen selection</td>
</tr>
<tr><td><tt class="docutils literal"><span class="pre">n</span></tt></td>
<td>Narrow selection</td>
</tr>
<tr><td><tt class="docutils literal"><span class="pre">F5</span></tt></td>
<td>Execute query selection</td>
</tr>
<tr><td><tt class="docutils literal"><span class="pre">Ctrl+L</span></tt></td>
<td>Set focus to URL bar</td>
</tr>
<tr><td><tt class="docutils literal"><span class="pre">Ctrl+K</span></tt></td>
<td>Set focus to query textbox</td>
</tr>
<tr><td><tt class="docutils literal"><span class="pre">Esc</span></tt></td>
<td>Clear query selection</td>
</tr>
</tbody>
</table>
</div>
<div class="section">
<h1>Implementation</h1>
<ul class="simple">
<li>Implemented entirely in XULRunner</li>
<li>UI is specified using XUL</li>
<li>All application logic in JavaScript<ul>
<li>Heavy use of jQuery library</li>
</ul>
</li>
</ul>
<div class="notes">
<p><a class="reference" href="http://developer.mozilla.org/en/docs/XULRunner">XULRunner</a> is an application framework based on Mozilla's Gecko layout engine. It comes with a lot of libraries for creating desktop applications. See the <a class="reference" href="http://developer.mozilla.org/en/docs/XULRunner_Hall_of_Fame">XULRunner Hall of Fame</a> for a list of prominent projects that are implemented using XULRunner.</p>
<p><a class="reference" href="http://en.wikipedia.org/wiki/Xul">XUL</a> (XML User interface Language) is an XML dialect for specifying GUI layout.</p>
<p><a class="reference" href="http://jquery.org">JQuery</a> is a very lightweight JavaScript library for doing Ajax-y stuff. In Cycloctopus, I use jQuery for these things:</p>
<ul class="simple">
<li>CSS query support</li>
<li>Determining the dimensions and offsets of DOM elements (this is not a trivial task)</li>
<li>Event handling</li>
</ul>
<p>JQuery works great in XULRunner! This is maybe not coincidence, as the author of jQuery, <a class="reference" href="ejohn.org">John Resig</a>, works at Mozilla.</p>
</div>
</div>
<div class="section">
<h1>I Love XULRunner!</h1>
<ul class="simple">
<li>Embedded browser</li>
<li>Cross-platform</li>
<li>Wide selection of widgets</li>
<li>Some cool non-GUI features</li>
<li>Java integration</li>
</ul>
<div class="notes">
<p>XULRunner runs on any platform that Firefox can run on. In fact, Firefox 3 is essentially a XULRunner app (you can also run other XULRunner apps through Firefox).</p>
<p>XUL offers a lot of standard widgets (check out the <a class="reference" href="http://developer.mozilla.org/en/docs/XUL_Reference">reference</a>). You can create your own widgets too.</p>
<p>One of the cool APIs you get is <a class="reference" href="http://developer.mozilla.org/en/docs/Storage">mozStorage</a>, an interface to SQLite! There is also support for cryptography, networking, SVG, XSLT, HTML editing, extension management, ZIP file creation, etc.</p>
<p>You can connect to Java code using <a class="reference" href="http://developer.mozilla.org/en/docs/JavaXPCOM">JavaXPCOM</a>.</p>
</div>
</div>
<div class="section">
<h1>I Hate XULRunner!</h1>
<ul class="simple">
<li>Documention is not good</li>
<li>JavaScript is inferior to Python</li>
<li>Directory structure is weird</li>
<li>Errors aren't displayed unless you enable the JavaScript Console</li>
<li>Deployment is not consistent among platforms</li>
</ul>
<div class="notes">
<p>I found quite a bit of useful documentation on the <a class="reference" href="http://xulplanet.org">XULPlanet</a> website. However, XULPlanet's references are not always up-to-date, and this sort of reference documentation really should be on <a class="reference" href="http://developer.mozilla.org/en/docs/Main_Page">Mozilla Developer Center</a>.</p>
<p>To be fair, JavaScript 1.9 is more enlightened than the JavaScript implementations in other browsers. You get list comprehensions, generators, and other ideas stolen directly from Python.</p>
<p>There exists a debug build of Cycloctopus that enables the JavaScript Console. There is also a way to <a class="reference" href="http://developer.mozilla.org/en/docs/Debugging_a_XULRunner_Application">embed the Venkman debugger</a>, but I don't understand how I'm supposed to use it once I embed it.</p>
<p>I don't much care for the directory structure of XULRunner. Instead of worrying about it, I put all my code and configuration into a single directory, and use a custom build script (written in Python), to create the proper directory structure.</p>
<p>The deployment story on Windows and Mac is quite different. See <a class="reference" href="http://developer.mozilla.org/en/docs/XULRunner:Deploying_XULRunner_1.8">http://developer.mozilla.org/en/docs/XULRunner:Deploying_XULRunner_1.8</a>.</p>
</div>
</div>
<div class="section">
<h1>Limitations</h1>
<ul class="simple">
<li>Can't view IE-specific pages</li>
<li>Query selections involving many elements is slow</li>
<li>DOM in browser might not be identical to DOM from urllib</li>
<li>Resizing the page messes up the selection</li>
</ul>
<div class="notes">
<p>Query selection is rather resource-intensive because every selection box consists of four absolute-positioned DIV elements. So if I select 100 elements, I have to create 400 additional DOM elements.</p>
<p>DOM nodes in the browser are often created/manipulated through JavaScript. These nodes aren't the same when you use <cite>urllib</cite> to download the page. So the XPath expressions you use in Cycloctopus won't always work when you switch to lxml.</p>
</div>
</div>
<div class="section">
<h1>Future Work</h1>
<ul class="simple">
<li>Fix annoying bugs</li>
<li>Display error messages for failed queries</li>
<li>Get lxml integration working<ul>
<li>Communicate with Python server using HTTP?</li>
<li>JavaScript-to-Python bridge?</li>
</ul>
</li>
<li>More features...</li>
</ul>
<div class="notes">
<p>One of the weirder bugs is that Cycloctopus crashes when you try to visit MSN.com. Fortunately, I do not have a strong desire to scrape from MSN, but it's still annoying. I think it's a bug in XULRunner, but I'm not sure.</p>
<p>In addition to displaying errors for failed queries, I should perhaps also display a "Details" link that you can click to get a detailed explanation of why a query failed.</p>
<p>PyXPCOM doesn't seem to be good choice, because I can't find any binaries anywhere, and also because Mozilla is supposedly moving away from XPCOM.</p>
<p>One possibility for the lxml integration would be to use PyQt 4.4 instead of XULRunner (version 4.4 will include a browser widget based on WebKit). I don't think I will go this route, but if XULRunner becomes too cumbersome to integrate with, I'll consider it.</p>
</div>
</div>
<div class="section">
<h1>Feature Glut!</h1>
<p>Some features I might like to add:</p>
<ul class="simple">
<li>Remember the last URL you visited.</li>
<li>Create a version that works as a Firefox extension</li>
<li>Add a JavaScript query mode</li>
<li>Windows installer</li>
<li>Mac application bundle</li>
<li>Generate the code to create site-specific Greasemonkey scripts</li>
</ul>
</div>
<div class="section">
<h1>Check It Out, Yo</h1>
<p>Source code, binaries, documentation:</p>
<p><a class="reference" href="http://cycloctopus.googlecode.com">http://cycloctopus.googlecode.com</a></p>
</div>

</body>
</html>
